The Automobile Dashboard or Instrument Panel (IP) as we know it today has evolved in the last 60 years or so, into a combination of control details such as knobs, switches, and sliders for actuation and selection of functions, together with analog or digital displays and light based tell-tales for communication of information to the driver. The actual provision of such components is governed in the USA by FMVSS (Federal Motor Vehicle Safety Standard) 101: Controls and Displays. In general, this standard specifies overall ground rules for the more critical functions operated by hand by the driver, leaving non-critical functions such as entertainment to the option of the manufacturer. Even some non-critical controls however, are specified as to their design goals, to avoid driver confusion. A particular stipulation of FMVSS 101 states that the Labelling (written or pictograph) for controls must be on or adjacent the control.
The Vehicle Instrument Panel is constrained as to available space for the instrumentation and control functions. Controls used by the driver cannot be so high as to obscure the drivers vision, or too low so as to require too much angular diversion of ones line of sight while in motion (the SAE guideline is 30 degrees max from normal line of sight out the windshield).
In a right—left, or cross-car context, the hand operated controls cannot be significantly behind the steering wheel used in present day vehicles, or too far to reach over to the passenger side by even the smallest driver using them.
In order to achieve added functionality made possible by modern electronics while at the same time reducing the ever increasing clutter of further controls required to achieve even present day functions, there is a big incentive to develop Instrument Panels which can fit in the available instrument panel space yet be reconfigured to serve multiple purposes, always with safe operation paramount. In particular, there is a need to provide such IP's which can not only be used on less critical control and informational functions, but for primary control functions as well, reducing cost and complexity and increasing justification.
Examples of prior art directed at some aspects of the problem are:
U.S. Pat. No. 6,246,935 by Buckley, Jun. 12, 2001 Vehicle instrument panel computer interface and display;
U.S. Pat. No. 6,344,793 Process for assisting a user of a motor vehicle . . . , Geck et al, assigned to Daimlerchrysler;
U.S. Pat. No. 5,757,268 Toffolo, et al. May 26, 1998 Prioritization of vehicle display features (on a reconfigurable display), assigned to Lear Corporation;
U.S. Pat. No. 6,373,472, Driver Control interface System, Palalau et al, assigned to Lear Corporation;
U.S. Pat. No. 5,539,429 Yano, et al. Jul. 23, 1996 Touch device panel, assigned to Mitsubishi;
U.S. Pat. No. 6,067,081 Hahlganss, et al. Method for producing tactile markings on an input surface and system for carrying out of the method, assigned to VDO;
U.S. Pat. No. 5,956,016 Kuenzner et al Operating device for Menu controlled functions of vehicle, Assigned to BMW;
U.S. patent application Ser. No. 09/963,565 Apr. 4, 2002 by Hirose, et al. Display device with screen having curved surface, assigned to Nissan;
PCT Patent application PCT/GB99/04006, Touch Sensitive Switch, Butler et al, assigned to Ford Motor Co.;
US pat app Kind Code 20020002432 Bockmann et al. filed Jan. 3, 2002 assigned to Volkswagen. Automobile multifunctional display and control device method;
In terms of the control of the vehicle, there are basically three major components (not including foot based controls).
Data Displays by instruments either analog or digital, and accessories. Vehicle data is generally speed, fuel level, engine rpm, battery charge, etc. Some of these are rapidly varying (e.g. RPM) others slow (eg. fuel). Some are more important than others.
Hand operated controls, generally today in the form of knobs, switches, sliders, levers and dials. Switches and knobs are most common today it seems. Most are located on the instrument panel, but some are on stalks protruding from the steering column, and window and seat controls and outside mirror controls can most often be found on the doors, or seats. Some controls (such as cruise control, auxiliary radio controls etc) can be found on the steering wheel in many vehicles.
Control labeling, today nearly universally achieved with printed data (language or pictograph) on or adjacent the control, to comply with FMVSS 101. In several cases, the data cannot rotate with the knob.
Modern technology has the ability to greatly expand the amount of data which can be provided, as long as the means exists to display it safely (legibility, position, etc). However, the means to interact with the data, if required or desirable has to be present in the controls, which at the same time need to comply with regulations. The steering wheel, which is a logical place for more control functions, is pretty much constrained today in this regard by the presence of the airbag, and the requirement for slip rings or other complex wiring
Central to the issue, is the question of just how the driver uses the controls?. I have observed that a driver of a vehicle today, interacts with the controls of the vehicle instrument panel, primarily in three ways:
By sight only;
By touch/feel only (possibly accompanied by a sound, such as a click-usually corresponding to a feeling sensation such as provided by a detent;
By a quick glance, and then touch—possibly followed by another quick glance to check a further or final setting of a knob, slider control, etc.
Feedback as to correctness of the action taken, if not obvious from sight or feel of the controls, is often times derived from the action of the device controlled itself, for example seeing and hearing the windshield wipers moving at high speed, when they were moving at low speed at a previous control setting.
Often while driving, only methods 2 and 3 above can be effected, as the control details may be too small to read or operate (radio function buttons are notorious) and/or require too much concentration (i.e. too many glances at the controls and their lettering, pictographs, and position) while driving to allow one to take ones eyes off the road for the time required to use sight only. Accordingly, for many users today, some functions are effectively inoperable while driving, since they cannot be worked by touch only due to the crowded nature of their placement, and they are too hard to see in a glance—especially for those who cannot easily correct their vision away from the far sighted vision needed to drive.
It should be noted that certain technical papers indicate that single glances of more than 2 seconds duration are considered unsafe, and some traffic safety researchers feel the limit to do any control activity should be just a few glances of no more than 1.2 sec. each. These studies indicate in considerable detail that excessive glance time can be related to ten's of thousands of deaths per year in the USA alone.
The new Auto Industry guide lines for “Glance time” suggest that control of the vehicle function should not require one to take ones eyes off the road for than 2 sec max, and no more than 10 glances (of 1-2 sec. ea) should be required to complete the operation in question. (while seemingly are a large number, some navigation tasks in certain vehicles have historically taken considerably more than this). This would seem the minimum stipulation one could possibly recommend in light of the suggestions in many technical papers. And it suggests that indeed there is considerable room for improvement to not only the situation today, but even what is being contemplated.
Not only are there problems seeing or touching small complex controls, some controls are not intuitive, and hard for the general public to understand in the manual, even if one has time to read and comprehend it. This manual comprehension problem is increased considerably, for those buying more expensive vehicles on short term leases. These vehicles are in addition, usually the ones with the most features, (and generally therefore the smallest buttons, due to instrument panel congestion), and many persons seldom take the time to fully digest the manual. (Some manuals today for feature oriented luxury cars are several hundred pages long !)
Generally a few functions, such as turn signal stalks, transmission levers, and some knobs are big enough and/or simple enough such that only touch is needed, combined with feedback from sensing the physical action enabled (e.g. wipers switched to high speed). Thus after some acquaintance period, most vehicles enable the driver to perform the basic functions by touch alone, or just a reasonably quick glance and then touch. However, this often applies only to basic operations, given the learning curve involved and the difficulty in operation of added features, such as complex HVAC (heating ventilation and air conditioning) systems, navigation systems, and “infotainment” devices.
The present situation thus greatly constrains the addition of still further vehicle related functions, made possible by computing, electronics and communication technology. Heretofore, the addition of such functions in some cases has been distracting to the driver and thus dangerous to the public. With fixed instrument panel space, much of which is taken up by airbags, glove compartments and the like, manufacturers have resorted to the above mentioned expedient of ever smaller and more complex controls in the remaining space, in order to gain functionality. Many are in addition located out of the drivers normal line of sight, and some even require one to stare at changing numbers in an awkward location. Others require reading very small print (often impossible for those requiring reading glasses while driving—a real dilemma for many drivers over middle age particularly).
Of late, many have shown interest in “Voice Recognition” as an answer to such problems, and increasing numbers of vehicles such as the Infiniti Q45 and Jaguar “S” model, have such incorporated. Using ones voice (and the listening to computer voiced data streams) in theory could reduce the need for glancing at all.
However, voice recognition has many problems. For example, to be useful, vocal prompts for the driver are often needed, which can be time consuming, frustrating, and difficult to hear. Undue concentration on same can also cause unsafe driving, and masks the presence of sirens. In addition, voice is both linear, and sequential. Only one message/action can be executed at once, and generally needs to follow a previous one. And the whole message has to be understood by the driver, or data can be lost and the process required to be repeated, which can cause driver frustration and endanger safety in extreme circumstances.
I have felt for some time that voice is not the answer. A recent study by Dr. Ben Shneiderman at the Human Computer Interaction Lab at the University of Maryland reported in The Washington Post, Thursday May 9, 2002, noted that voice commands also require much more precious brain activity than visual “eye-hand” tasks. Thus voice cannot be good for driving safely. The Instant invention however, maximizes the efficiency of such Eye-hand tasks, making them familiar and easy to see and act upon.
Voice aside, perhaps the only way more data can be presented to, or received from, the driver is through the use of specialized displays and entry devices which can be reconfigured to suit the need of the user at the instant of use. Such displays are commonplace today in the computer world (e.g. “Windows”, “Web Browsers”), so why not in a car?
Some companies have proposed or developed conventional reconfigurable computer “windows” type systems using up/down/left/right buttons or even joysticks to select the various software presented screens and menus thereon in an attempt to solve this problem. Such devices are however, not intuitive and hard to use, for at least 95% of the populace I feel. Because of such problems, even where implemented, they have been relegated to non-critical functions, such as navigation, climate control, cell phone dialing or audio system entertainment. And even here, they are intrinsically unsafe in many manifestations. There is little or no tactile feel, and too much reliance on consecutive visual recognition of displayed items—which in addition are often small in size, as noted above.
Problems with computer displays of the multiple windows type used in vehicles, are discussed in U.S. Pat. No. 5,995,104 by Kataoka, aimed at a Navigation system. Some opportunities available with computer based data presentation in visual and multimedia form are discussed in Obradovich, et al U.S. Pat. No. 6,009,355.
It should also be noted that the lack of natural intuitive tactile feel further makes such menu driven “pure computer screen” type systems slower to operate and constantly requiring visual concentration. This is an issue when the control system is used for any time-critical tasks and has obvious safety connotations.
This is not just true in cars, but in other stressful situations as well, for example in military vehicles, construction equipment, and control of home appliances by a harried housewife in the kitchen.
Another approach to reconfigurability is a touch screen, which is widely regarded as one of the most intuitive of computer interfaces. To my knowledge, the application of touch screens for computer input in automobiles was first achieved in the 1988, in the General Motors Buick Riviera using a relatively small CRT display located in the center stack, in a relatively low position. Such a display is described in U.S. Pat. No. 4,787,040 by Ames, et al. entitled Display system for Automotive Vehicle issued to IBM Corporation Nov. 22, 1988.
However, widespread employment of such touch screens has historically has been limited because it cannot be operated by feel, and requires of a driver more than a quick glance to operate successfully. This is in part I believe because there is nothing on a conventional touch screen to reference ones self to, except the displayed data itself—which then has to be continually read.
To combat this problem (experienced in the early Buick display and others), some vehicles such as the Cadillac CTS have incorporated ordinary non-touch screen LCD displays, with buttons and other conventional touch type controls around the periphery thereof to effect reconfigurable operation in conjunction with icons displayed on the screen nearby (and thus to a degree meet FMVSS 101). Taking this a step further, the Denso Company has worked with Jaguar to introduce a touch screen of the LCD type in the their X model. It has a small 7 inch screen, also surrounded by conventional buttons and touch controls, and located in the center stack. But the display itself has no reference and no feel, and the surrounding buttons are, as in many other examples, small and hard to read. Thus the fact that certain touch icon functions on the screen are reconfigurable is only a small improvement on the previous and prevailing situation.
Another example of a touch type data entry device is that of U.S. Pat. No. 6,067,081 by Hahiganss, et al., assigned to VDO corporation of Germany, an Automotive parts manufacturer. This is a device providing programmable sensations, which can be used in conjunction with a display screen, though there is no teaching that the touch is of the display screen itself, desirable for intuitive interaction. Thus the Hahiganss et al invention would seem not to comply with FMVSS 101.
Palalau et al, U.S. Pat. No. 6,373,472 includes switches on the steering wheel with a heads up display on the windshield. This is a novel approach which offers many advantages, but requires added cost and complexity, and a departure from the customary vehicle operation that drivers are used to. In addition it could be dangerous in some cases as the more fumbling with the wheel that occurs, the more the possibility of changing vehicle direction as a result. And too, the wheel, with its airbag and rotary motion, severely limits the amount of sophistication one can put in the controls thereon.
A recently published PCT application PCT/GB99/04006 by Butler et al of Ford motor company illustrates a multifunctional switching device for the operator to use whose groove aspect which bears some similarity to certain tactile groove aspects of the invention, though it is neither a touch screen nor reconfigurable per se. Like some embodiments of the instant invention, it can provide data only when needed by the user (“secret until lit” feature), but it cannot be reconfigured as to what this data is.
An alternative control device for a car having sensations of feel is a force feedback joystick coupled with a conventional (and programmably reconfigurable) display, such as an LCD based display, on the instrument panel. One example, called “I-Drive”, has been introduced by BMW in its 7 series car and appears to be disclosed at least in part in U.S. Pat. No. 5,956,016 of Kuenzner et al of BMW and in U.S. Pat. No. 6,154,201 assigned to Immersion Corporation, which describes various force feedback of a conventional type to a knob useful for automotive or other use. This has some similarity to certain “programmable feel” related aspects of the present invention and its predecessors, even though the knob disclosed is not located on the screen, nor inter-related with touch characteristics of the screen as is the instant invention.
As a general comment, I-Drive does not appear to be intuitive to use, and has received considerable criticism in the Automotive press (see for example, Road and Track, issue of May 2) The manufacturer must have realized this, as in the 7 series manifestation at least, many of the functions are backed up with conventional controls. as well, thus adding cost. This also may be because the FMVSS 101 regulations require the markings of a control to be on or adjacent the control, and the joystick device by itself cannot meet this criteria.
If one can provide a reconfigurable instrument panel that can be safely used, then U.S. Pat. No. 5,757,268 Toffolo, et al. (referenced above) gives a good description of some of the opportunities available in optimizing the presentation of displays and the information provided. Toffolo however, is focused on a conventional display screen approach and is generally limited relative to the display size, two areas over come by the instant invention. And Toffolo does not address several control areas of considerable importance, such as designation or confirmation by the driver of data, or the overriding of important controls (e.g. speedometer) by visual or other data in the case of emergency situations.
Finally there are many projects around the globe aimed at sensory equipped intelligent vehicles and highway systems. Representative patent applications in this area are Ser. Nos. 09/963,490 Road lane marker recognition by Furusho et al, and 09/951,499 Lane recognition apparatus for vehicle, Shirato et al, both of which are assigned to Nissan, and utilize TV camera and machine vision technology to help guide a car such that it stays within lanes. Some of this technology however is very difficult to perfect in a full automatic mode. I know of no reference which has addressed the issue of manual assist to such systems, provided in a large screen tactile control and display device of the type disclosed herein.
To conclude, no prior art reconfigurable instrument panels known to me provide any feel on a display screen itself, as to where on the screen the users finger is, nor do they provide a method for tactilely signalling information back to the user data which would make reliance on long and dangerous glances unnecessary
In addition, no prior art instrument panel uses common knobs and other selection and control details familiar to the driving public of today which can be reconfigured either physically or in terms of their displayed function, data or other variables in a manner that appears to intrinsically comply with FMVSS 101.
Furthermore, no prior art instrument panel I am aware of provides means for safely displaying real time or down loaded video data to the driver in a large enough to easily to see in a cost effective manner, vital for observation of critical events inside and outside the vehicle, as well as to make downloaded information from remote sources easy to see at a glance. And in the same vein, no known instrument panel is capable of providing large lettering for all necessary controls to aid the elderly or vision impaired driver.
And in the same vein, no prior art reconfigurable instrument panel to my knowledge, and even many controls of conventional IP's, do not utilize controls which can easily be “hit” or otherwise actuated, with just a helping glance or by feel alone. And, no prior art teaches incorporation of both knobs or other control details and tactile touch functions on an instrument panel to allow one to optimally execute functions in concert.
In addition to the prior art referenced above, there are a series of related US patents by Denny Jaeger, some of which are in conjunction with Kenneth Twain (eg U.S. Pat. Nos. 5,572,239, 5,841,428, 6,326,936) which relate to techniques for implementing control devices such as knobs and switches on flat panel displays (electroluminescent or LCD) to achieve the benefits of controls function reconfiguration. The approach taken however, results in complex (and expensive) systems which, while programmable as to the display, do not facilitate interchange of the physical components. In many embodiments too, they require significant optical compromises in the display (such as blank spots) and the flat panels used require specialized busbars removing these displays from the mainstream activity. In some cases, intolerable leakage problems can occur as the LCD liquid crystal material can leak around the knob shafts for example.
To solve the problems encountered, some embodiments (of the many shown) of Jaeger and Jaeger et al., have attempted to use optical techniques to shoot through the screen so to speak. In the earliest example, a detector and source for each device such as a knob or switch, is located and attached to the other side of the display. It is positioned to shoot through either a specialized transparent busbar or a bus bar routing hole specially created in the display. This can work presumably, but is complex and expensive, and would appear to use up valuable display space, as well as requiring a device dedicated and positioned to operate each physical control. In LCDs requiring separate specialized backlight sources for their operation, such sensing devices would appear impractical, as they would block the radiation from the backlight.
Where optical sensing is employed in later patents by Jaeger and Jaeger et al, these inventors have apparently focused on having the sensing device (eg a phototransistor) on the front of the display, located on a member attached to the screen in some way. Either the light source is also on the front, or light from the display is programmed to interrogate the position of the sensor on the knob which senses this activity. The inventors also have disclosed sensors of this type which are largely incremental, rather than absolute, which limits some control regimes and can cause errors. Additionally, Jaeger has disclosed a small LCD device electrically placed on the front of a knob or button and controlled to change information thereon, an approach which requires specialized wiring and LCDs for every physical control detail used.
Finally, Jaegers invention are specific to flat panel displays and cannot provide curved or irregular display capability of maximum utility and style for Automobile instrument panels, and teach little or no ability to use materials other than glass as the display surface, also a requirement in vehicles (for passive safety and other reasons), especially as the displays grow in size.
The instant invention solves all of these problems, by using an electro-optical detection means, typically an inexpensive TV camera, to sense, typically from the rear of a rear projection display of nearly any material and shape, more than one physical detail and its position, and generally its absolute position as well. And it can operate on any desired combination of physical control details (such as knobs sliders etc) with only a software change. This then enables the complete interchange of the display and control surface, including any control details such as knobs, which are simple passive devices with no wires needed. In addition, the invention herein, unlike the prior art, allows an integrated touch screen capability to be provided which can be completely integrated with the physical control capability, in some cases even at no added manufacturing cost. The Instant invention is thus arguably of higher performance, simpler to maintain, and lower in cost—especially critical for high volume applications such as automobiles.